Using Early Quizzes to Predict Student Outcomes in Online Introductory Biomedical Informatics Courses

Identifying students at risk for poor performance in large online classes can be challenging. We determined whether the first few quiz scores can be used to identify students who will have poor course outcomes in an introductory informatics class. Mean scores on the first four quizzes can identify students at risk for failure. Even the first quiz score significantly predicted introductory informatics course outcome. Automating early identification of students likely to fail may allow instructors to create targeted interventions. Introduction: Distance learning enabled by the Web is one approach to extend the geographic reach of existing informatics training. Regardless of their career goals, many students start with an introductory informatics course. As a result, more introductory informatics courses are offered online to ever increasing numbers of students with diverse backgrounds and career goals. As the number of students enrolled in each course increases it becomes more difficult to offer personal assistance to every student. If “at risk” students can be automatically identified early, instructors may be able to intervene and rescue students who otherwise would earn a failing grade or drop the course. We determined that the first four weekly quizzes could be used to identify students at risk for failing or dropping an introductory informatics course within the first month of the semester. Methods: We reviewed data from two online introductory informatics courses presented at the University of Texas School of Biomedical Informatics (UT) at Houston and the Medical Informatics program at the University of West Florida (UWF). For UT we used all students beginning with Spring 2007 and ending with Spring 2010, for a total of 205 graduate students. For UWF we used the Fall 2009 semester with a total of 42 students, 28 undergraduates and 14 graduates. We built the prediction model based on UT data and then validated the model using UWF data. We compared automated identification of students at risk for failure to manual identification by a faculty member familiar with the course design and content (TRJ), but not aware of the individual student outcomes. Results: We considered students who scored an average of 75% or below on the weekly quizzes to be at risk of NSC. We chose 75% as the quiz score threshold because this was the highest score where the predictor’s false positives were equal to its false negatives. Even the first quiz score is a significant predictor of course outcome. Each successive quiz added to the model improved the model’s performance. ROC = Receiver Operator Characteristic, PPV/NPV = Positive/Negative Predictive Value. Discussion: While representing only 6% to 8% of the total grade, the first four quizzes are highly predictive for course outcome. Using only the first two quizzes available by the UT add/drop deadline still allows prediction, but with a lower PPV. Conclusion: A simple threshold model developed at UT predicted non-successful completion at another institution offering an introductory course, UWF. Automated prediction compares favorably to human instructor prediction. Acknowledgements: The authors thank all instructors and students who contributed data to this study, including Stephanie Reedy at UWF for help with data collection. This work was funded in part by NCATS Grant UL1 TR000371 establishing the Center for Clinical and Translational Sciences at the University of Texas at Houston. Week(s) Area under ROC Curve (AUC) PPV